Prom: a Exible, Prolog-based Make Tool

Maintenance of large, portable software systems often leads to requirements which cannot be solved by the traditional make tool. Abstract naming schemes for les and programs, as they are used by preprocessors in actual make tools, are fundamental for a more general solution. But, as shown in this paper, a preprocessor is not powerful enough for all requirements. Some kind of \database" with the information for making speci c les is needed. Abstract names looking very close to Prolog terms and the need for a \knowledge base" lead directly to the idea of having a Prolog interpreter doing the le update job. As a prototype, prom has been implemented which is introduced at the end of this paper. Introduction The classical UNIX tool make, as introduced in [Fel79], is commonly used for compiling a program which consists of a couple of source les. Furthermore, all of the les are normally located in a single directory and are dedicated to be compiled on one single machine. For this kind of application, make has su cient power. But with growing software systems and with the high connectivity of computers, make reaches its limitations whereas the principle idea of having a program to update les by a set of relations between them is still powerful enough. Because of this reasons an improved make tool was designed and implemented. The requirements of large, portable software systems The special requirements of large software systems originate in their size. This kind of program is too large to t into a single directory following the intention of grouping only several related les to keep the overview. In designing a software system, one usually divides it into several abstract components, like classes of objects or modules. We consider classes instead of single design components because we need multiple implementations for di erent target machines or with di erent behaviours. As general term, we concern modules as a concept to support information hiding as de ned in [Goo73]: \Modularity denotes the ability to combine arbitrary program modules into larger modules without knowledge of the construction of the modules." In the following, we deal with so-called \module classes". They are de ned as the set of all possible implementations which ful l the speci cation of a module. Although these sets have an in nite number of elements, make tools only treat those les which represent a module class in form of its interface speci cation and its existing implementations. A brief description of a class hierarchy for software development can be found in [Sch89]. Module classes may be divided recurrently. This leads to a \consisting relation" between the components of a software system describing which component consists of which ones. Figure 0.1 shows an example of a software system consisting of sample module classes. Under the guideline of information hiding, the module classes are separated and interact only using their interfaces. An obvious way of separating module classes is to transform the graph of the consisting relation directly into a tree of directories. Each directory contains exactly the modules which build its module class. Each module class is accompanied by its make le, in which the relations between its les are collected. So we can change the implementation of one part of our software system without a ecting the others. As we consider our software system to be portable, we also want to \port" our make les with a minimumof changes. A rst step towards this aim is done by make with the provided macros. So we do not need to take care of names and command line options of the standard compiler tools. But this is not su cient. What we need is a complete abstraction from command line syntax and le naming conventions. The only information in a make le should describe relations between the involved les. Therefore we need a notation in the make les which allows us to mention les completely abstracted from the names required by the underlying operating system. This notation is the key for keeping multiple versions of a software system in a single directory tree. An example is a UNIX machine with the directory tree of a software system on one of its le systems. This le system may be visible (for example via NFS) on several other UNIX machines (like System V, BSD, SunOS, AIX,: : :). For assembling on all di erent machines we only need to support a speci c mapping from the abstract notation in our make les to the machine-dependent names and syntactical conventions. 1 system HHHHHj module module : : : class 1 class n = ZZZ~ module module : : : class 1:1 class 1:m module class 1:m:1 Figure 0.1: The module class hierarchy of a software system. This also works with more heterogeneous operating systems, for example AIX and MS-DOS which may share a le system via DOS Merge. And even if there are target machines \o line" from the source directory tree, we can assemble programs using this method. We only need to transport the source les (including the make les) initially. As we have seen, a make tool should supply an abstracted view on the components of a software system which hides the system-dependent platform. In the next section a rst approach using a preprocessing tool is described. Using a preprocessor In the following, we consider cake as a make tool which uses the C preprocessor for its input les. cake provides us with several additional features like transitive and dynamic rules. But the main advantage arises from the C preprocessor, whose macro facilities are the key to the abstracted notation mentioned above. For a brief description of cake and its use for portable software systems see [Som87] and [Kie90]. Qualitative improvements with a preprocessor In the following, we will see how to use the C preprocessor to abstract from system dependent notations. First, we can use simple macros similar to those known from make. The following gure shows some examples. For instance, using one of the rst two lines allows to hide the C-Compiler currently in use. CC cc CC preprocessor /usr/local/GNU/bin/gcc CCFLAGS -traditional -O -DBSD YOPTS -d 2 Parameterized macros are much more powerful. They allow to classify les. In the following, we speak of le classes|the sets of les denoted by these parameterized macros. Some very common examples are: C( le) le.c OBJ( le) preprocessor le.o EXE(program) program VER( le) le,v While exclusively using these names, one can easily state relations between the les in use without mentioning any machine-dependent names. Looking at the example of the network global le system from which one source code should be assembled for several di erent target machines, name con icts must be resolved. For this purpose, we only need to provide the C preprocessor with the right macro de nitions. For example, we de ne OBJ( le) as le.o-sun3, le.o-sun4 or le.o-hp, depending on the target machine. Such C preprocessor macros are only helpful if de nition and utilization is separated in di erent les. In principle, there are two sets of les in use: On one hand, there are counterparts to traditional make les located in the source code directories. Here, the information is placed concerning the les realizing the corresponding module class. Sub-module classes are also declared here. On the other hand, there are les containing the mapping from abstract (e.g. le class) names to system-dependent syntactical forms. These les are located in a so-called \system directory". Such a directory must exist for each target machine. The appropriate one will be selected when starting cake by specifying a search path for the C preprocessor. In addition to the le classi cation we also have to describe the relations between the les. These relations declare the creation of les out of each other. Examples are: LINK( target, list of object files, options, libs ) MAKE ARCHIVE( archive, list of objects ) Those relations are used as part of le creation rules, as they are known from make. Here, the relations form the actions which have to be performed when the rule is triggered. The actions are written as program calls. As they are system-dependent, their declaration also occurs in the system directories together with the declaration of the rules. Problems which cannot be solved by a preprocessor On an idealized view, all information needed to create a program out of its sources is localized in the system directories. This works well, as long as there are only \default" actions to be performed. So the special cases have to be expressed in the source le directories. But this causes problems which can be seen from the following examples. We consider le generation rules in the syntax known from make like target : list of sources list of actions 3 The most obvious \exception" from default rules is the creation of an executable out of object les. Their number is unknown and their names are usually completely di erent (independent) from the name of the executable. A simple rule for this could be: EXE(program) : OBJ(one) OBJ(two) OBJ(three) LINK( EXE(program), OBJ(one) OBJ(two) OBJ(three) ) This rule cannot be stated as a default for the reasons mentioned above. To overcome this, one might propose the following modi cation: EXE(program) : OBJLIST(program) LINK( EXE(program), OBJLIST(program) ) Using this rule, one has to declare OBJLIST for each executable in a source directory make le. But this is impossible, because the preprocessor avoids the rede nition of its macro OBJLIST from the second de nition on. This is because a preprocessor interprets such a de nition as a rule for pattern substitution instead of a \Prolog fact" like declaration. One could also think of forming individual macro names for each executable. This looks like EXE(program) : OBJLIST program LINK( EXE(program), OBJLIST program ) and define OBJLIST example OBJ(one) OBJ(two) OBJ(three) But this approach fails because of the two-step evaluation. First, the preprocessor transforms the rule into: program : OBJLIST program cc -o program OBJLIST program So the intended match between OBJLIST program and OBJLIST example cannot be performed afterwards in the update process of the make tool. This phenomenon also a ects other le classes. For the reasons stated, it is impossible to set compiler switches for individual les only. Every time a le has special requirements, one has to copy the entire set of rules for all target machines. But that kind of information was intended to be located in the system directories. This violates information hiding between the software system and the specialties of the target machines and leads to severe update problems for the make les with growing size of software systems and the number of supported machines. Using a Prolog interpreter as a make tool From the experience with the previous approach we conclude that entries in make les should have declarative nature. These declarations should be entered into a knowledge base. Together with a powerful set of default rules, this knowledge base is an adequate basis on which le updates can be performed. The abstract notation for les, programs, and actions which was introduced together with the preprocessor concept looks very close to Prolog-like terms. The terms inside the default rules have to be uni ed with actual lenames. Prolog's term uni cation facility is an excellent way to do this. Unbound variables in arbitrary positions of terms provide us with almost any degree of exibility in formulating le creation rules. 4 Feasibility of implementing make with Prolog The reasons mentioned above suggest Prolog as a predestined candidate for the implementation of a make tool. A closer look at the demands of such a tool will show that Prolog is indeed powerful enough to perform all necessary tasks. The complete set of requirements for updating les out of each other is the following: Read les. For reading make les Prolog o ers the predicates see and read. Compute the actions to be performed. A kind of \update algorithm" can be formulated as Prolog predicates. Execute external programs. For this purpose, the standard predicate system can be used. The Prolog interpreter should take the exit code of a program to succeed or fail with system. If it does not take the exit code, one has to ensure otherwise that defective les will always be deleted. Get information about timestamps of les. This is the only problem which has to be solved outside the Prolog functionality. Possible solutions are (in order of increasing execution speed): 1. An external program (called by system predicate) writes the timestamps into a le which can be read by the Prolog interpreter. 2. An external program delivers a return value which is interpreted as a timestamp. But interpretation of such a return value must be supported by the Prolog interpreter. 3. A builtin predicate timestamp will be added into the Prolog interpreter. This requires the source code of the Prolog system. 4. TheProlog interpreter may o er an interface for executing C code as part of the Prolog program. A Prolog-make's knowledge base The core of a Prolog-based make is its knowledge base. It is created out of make les and of timestamp information of the les constituting an application. Figure 0.2 shows an overview of the information involved in the update process which is also described in the following text. The names of the le classes used in the gure only serve as examples without profound meaning. One major part of information in the knowledge base is extracted from two external sources: system make les Located in a so-called system directory, they declare terms to map the abstract notation onto the requirements of the target machine. They form the system-speci c part of the knowledge base. application make les Located in the source code directories, they declare terms concerning les of the speci c application. They form the application-speci c part of the knowledge base. Applying the update algorithm, the status information is collected. It consists of le attributes like existence and update time. The algorithm operates in two phases: The rst phase creates a le-dependency graph out of the knowledge base. Unlike the consistinggraph of the application, this graph is acyclic but in general no tree. 5 The second phase applies creation rules beginning with les without successors in the dependency graph, using its topological sorting. The knowledge base is constituted by the following Prolog predicates which represent the information mentioned above: create File creation rules, like the ones known from traditional make, consist of a target le, a list of source les, and a list of actions to create the target out of the sources. depend With this predicate, dependencies between les are declared. This is needed if additional source les which are not mentioned in a creation rule are used. An example is a header le included by a C program. define These de nitions describe the expansion of terms. exists This predicate succeeds if a le exists. timestamp This predicate represents the relation between les and their update times. isok This predicate succeeds if a le is up to date in the sense of the make les in use. It is needed, because the le-dependency graph is not a tree. Every le will be marked as isok after its creation. So intermediate les are created only once per update process. The realization of prom The concepts lead to the implementation of prom. We start its description with the de nition of the syntax of make les interpreted by prom. Syntax of make les Unexpectedly, the make le format became a critical design feature. In a rst approach, the syntax chosen was very close to those les interpreted by traditional make or cake. But this leads to severe runtime problems because of the character-oriented interpretation. The second version, as introduced in the following, was based on Prolog terms. It can be easily interpreted by the standard predicate read. The syntax is a compromise between interpretation e ciency and familiarity with the make syntax. The latter is achieved by prede ned operators which allow to form terms looking close to traditional make le entries. These operators are: create, default, define, depend, include, search, and `:'. As a side e ect, orientation on Prolog terms gives more freedom in writing make les. Whitespace characters can be inserted for clarity of structure. They are no longer crucial like newlines and tabs in traditional make les. Figure 0.3 shows the make le syntax. It is a context-free grammar, known from Prolog, with terminal symbols enclosed in brackets. The start symbol is entry. As a convention of read, each term has to be completed by a dot (`.'). 6 ................................................................................. .................................... .......................................................... .......................................................... ....................................... ....................................... application ? ? system module module make les : : : class 1 class n ? ? .................................... module module : : : class 1:1 class 1:m ? module application class 1:m:1 ............................................................... make les ..................................................... ..................................................... ................................................................................. ......................................................... .................................................. ? .................................................. ................................................................................. .................................................. .................................................. application speci c EXE(a) declarations ? ? ? OBJ(b) OBJ(c) system speci c ? ? ? ? ? de nitions / default rules C(b) HDR(b) LEX(c) HDR(c) status of ? ? ? update process CWB(b) LTX(c) knowledge base ......................................................... le dependency graph ................................................................................. ............................................................... Figure 0.2: Information involved in the update process with sample application make les and ledependency graph. Semantics of make les prom processes make le entries of the types listed below. 1. define: With define, the expansion of a term is speci ed and entered into the knowledge base. Terms are expanded to lists of terms. 2. default define: This de nition will only be entered into the knowledge base, if there does not already exist any other de nition for a term matching to Term. This feature is useful for defaults in system make les which will only be used if there is no corresponding de nition in application make les. 3. depend: With depend, one declares that a le called the \target le", depends on a list of other les. These les are treated as sources, independent from the creation rule chosen for the target le. The target le's sources are the source les mentioned in the creation rule united with the ones from all depend declarations matching the target le name. 4. create: create declares a creation rule for a target le. It implicitly declares a set of source les for this target which will be used if this rule is triggered. The list of actions declares which programs will be executed using this rule. 7 entry --> [define], term(Term), [=], terms(Expansion). entry --> [default], [define], term(Term), [=], terms(Expansion). entry --> [depend], term(Target), [:], terms(Sources). entry --> [create], term(Target), [:], terms(Sources), [-->], actions(Actions). entry --> [include], term(Filename). entry --> [search], [include], term(Filename). actions(A) --> action(A). actions([A1|Others]) --> action(A1), [','], actions(Others). action(A) --> term(A), f A =.. [call| ] g. terms(T) --> term(T). terms([T1|Others]) --> term(T1), [','], terms(Others). term(T) --> constant(T). term(T) --> structure(T). constant(C) --> [C], f atom(C) ; var(C) g. structure(S) --> [S], f S =.. [Functor|Arguments], Functor n== ',', Arguments n== [] g. Figure 0.3: Make le syntax of prom. Actions must be terms in the form of call(: : :,: : :) with arity greater or equal to one. 5. include: The content of a le will be inserted in this make le. 6. search include: Like include, but the le will be searched through a prede ned search path. Using this feature, the appropriate system make le for the target machine will be included. There are three special features which can be used in define entries. They are based on capabilities of the term expansion facility of prom. 1. concatenation The `+' operator concatenates its arguments to one single atom. 2. external evaluation The functor eval interprets its structure components as an external program call. The output of the program called is taken as term expansion. 3. empty expansion A very special term expansion feature is its behaviour in the case of an unde ned term. In general, \unde ned" means the impossibilty of matching the term with a knowledge base entry. Unde ned terms will be expanded to nothing. This feature is only applied to terms forming structures. Atoms which cannot be matched remain unchanged. For example, one may use the term compiler flag(program) which will only be expanded for les in need of such a ag. 8 The underlying Prolog interpreter C-Prolog, Version 1.5, was used for implementation. This is a small but fast interpreter for 32 bit UNIX machines available in source code. This interpreter was enhanced by a builtin predicate timestamp for retrieving timestamps of les. This could be done very easily with only few changes to the source code. prom was tested on HP 9000, Series 400 and PCS Cadmus 9600. Because C-Prolog claims to run on all 32 bit UNIX machines, prom can be used on this set of platforms. The startup mechanism The aim is to build a stand-alone software tool. So we want the Prolog interpreter to automatically start with processing make les and updating target les. C-Prolog o ers a mechanism using a startup le which can automatically be executed after invocation of the Prolog interpreter. We use a startup le which rst de nes the special operators, then reads make les and nally updates a target le. The name of the target le is read from a second startup le which is accessed prior to the make les. All command line options are passed to prom through this second startup le. Especially the system-dependent search path is transferred this way. So prom uses the appropriate mapping from its abstract names to system-dependent syntaxes. The implementation prom has been implemented under the paradigm of literate programming, as it was introduced in [Knu84]: \Instead of imagining that our main task is to instruct a computer what to do, let us concentrate rather on explaining to human beings what we want a computer to do." Programming literate, one writes documents in which program code and documentation text are integrated. A so-called WEB system extracts the program source out of these documents for machine interpretation. In absence of a Prolog-WEB, we used Makeprog to write code for prom. Makeprog, as introduced in [Sch90], is a generic WEB processor without language-speci c features. It can be used for arbitrary languages. The source code of prom is structured as shown in Figure 0.4. It consists of three major components with functionalities described in the following: pmake HHHHHj ? make le update term scanner algorithm expansion = external evaluation Figure 0.4: Internal structure of prom. 9 make le scanner The scanner reads make les, performs directory searches, checks the input syntax and transforms information into the knowledge base. update algorithm The update algorithm works as described above. It creates a le-dependency graph for the target le and triggers necessary actions. term expansion The term expansion facility performs transformations from terms into lists of atoms which represent literals in system-dependent syntaxes. A separate subcomponent is the interface to external programs which can be used to evaluate special terms. Making use of prom prom has been used for maintenance of several software systems. It has been found appropriate for large applications which have to be maintained in several versions and for several target machines. Multiple search paths allow system-wide and user-speci c \system make les" to tailor adequate sets of rules and de nitions for any kind of application. In the following, principles of make le writing for prom are treated on examples. Our rst example simply shows de nitions. We de ne the name of a program with cc cmd. obj(File) de nes the naming convention used for object les. Notice that File is an uninstantiated variable! While expanding this term, File's current instantiation is concatenated with `.o' using the `+' operator. define cc cmd = 'gcc'. define obj(File) = File + '.o'. The second example shows an overridable default de nition. An application-speci c make le may contain a line with the unconditional de nition. This line must be read before the system make le (with default define's) is included. default define project dir = '$HOME'. define project dir = '/usr/foo/bar/application'. Now, we consider an example of a le creation rule. Here, an executable le will be created out of corresponding object les. The application make le contains the application-dependent de nition of objlist. The last two lines of this example show the executed command when this rule is triggered for exe(application). As an unusual example, the \depend" line declares the dependency of each executable le from the make le. create exe(File) : objlist(File) --> call( cc cmd, objlist(File), '-o', exe(File) ). define objlist(application) = obj(one), obj(two). depend exe(File) : makefile. exe(application) =) gcc one.o two.o -o application 10 The last example shows two features: First, eval(: : :) calls get includes which outputs namesof les included by c(File). These les are added to the le-dependency graph as sources ofobj(File).Second, cc flags is an example of le-speci c compiler switches. The creation rule containscc flags(obj(File)) as a \hook". The last four lines show the commands executed for one lewith cc flags de ned (obj(file1)), and one le without (obj(file2)). Notice that cc opts isunde ned for all les in this example!create obj(File) : c(File), eval( get includes, c(File) )--> call( cc cmd, cc opts, cc flags(obj(File)),'-c', c(File), '-o', obj(File) ).define cc flags(obj(file1)) = '-I/usr/include/X11', '-DBSD'.obj(file1)=) gcc -I/usr/include/X11 -DBSD -c file1.c -o file1.oobj(file2)=) gcc -c file2.c -o file2.oFuture workThe following problems remain when working with prom. There may occur collisions of prede nedProlog operators with term literals. For example, it is impossible to have a directory include,because of the operator having the same name. A possible solution may be the rede nition of thesyntax avoiding any operator. This would lead to \traditional" Prolog terms like define(a,b)instead of define a = b. But this modi cation reduces readability of make les.Another inconvenience comes from commonly used UNIX shells. While entering the name of thetarget le, one has to express it in a term. The parantheses used here are interpreted by the shell.So one has to quote target lenames, like pmake "exe(application)".Further developments should improve robustness and error handling. Prevention of in nite loopsand treatment of contradictions in make le declarations belong to this eld.The list of \user wishes" also contains some debugging features. As an example, one can thinkof making the le-dependency graph visible to users. Other debugging information like rule identi-cation could be useful, too.AcknowledgementsI wish to thank the members of System-Programming-Group at TU Darmstadt for many helpfuldiscussions; especially Prof.Waldschmidt for reviewing earlier versions of this work and MarionDirscherl for improving the English version of this text.11 Bibliography[Fel79] S. I.Feldmann: Make|A Program for Maintaining Computer Programs. In Software|Practice & Experience. Vol. 9, No. 4 (1979), pp. 255{265.[Goo73] G.Goos: Language characteristics. In F. L.Bauer, editor, Advanced Course on SoftwareEngineering , Vol. 81 in Lecture Notes in Economics and Mathematical Systems, chapter 2,page 54. Springer, 1973.[Kie90] T.Kielmann: \Cake fur MS-DOS". Semester Project, Institute of Theoretical ComputerScience, Technical University Darmstadt, September 1990. (In German).[Knu84] D. E.Knuth: Literate Programming. In The Computer Journal , Vol. 27, No. 2 (1984), pp.97{111.[Sch89] J. Schrod: \YADD|Yet Another DVI Driver Family". Diploma Thesis, Institute of The-oretical Computer Science, Technical University Darmstadt, January 1989. (In German).[Sch90] J. Schrod: \The MAKEPROG System". Technical Report TI-3/90, Institute of Theoreti-cal Computer Science, Technical University Darmstadt, 1990.[Som87] Z. Somogyi: Cake|A fth generation version of make. In Australian Unix system UserGroup Newsletter , Vol. 7, No. 6 (April 1987), pp. 22{31.12